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An apparatus and method for producing composite data by 
a server computer involves deforming a template creating a 
mapping relationship between co-registcrcd data and subject 
data, filtering the co-registered data, and mapping this 
filtered co-registered data according to the mapping data. A 
client computer requests the composite data from the server 
computer and transmits the subject data to the Sjerver com- 
puter. The cheot presents the received composite data to an 
operator and monitors the operators use of the composite 
data. 
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METHOD AND APPARATUS FOR 
PRODUCING AND ACCESSING COMPOSITE 
DATA 

BACKGROUND OF THE INVENTION 

The present inveation relates to information systems and 
methods, and more particularly to data fusion systems. 

Many applications require the contemporaneous assimi- 
lation of large amounts of data. Medical, military, and 
commercial applications require operators to engage in 
procedures and make decisions based on data describing 
various subjects represented by, for example, images, 
recorded sound, and text Current technology does not 
present personnel with a unified view of this subject data to 
allow them to use all available data to make informed 
decisions. 

For example, a physician providing medical treatment, 
reviews image data acquired in multiple modalities, such as 
magnetic resonance ("MR"), computed tomographic 
("CP*), and X-ray images, medical journals describing 
procedures, video images, such as ultrasound, and atlases 
describing anatomical structures. A physician must therefore 
consult several sources to review the data necessary to 
provide patient treatment. These sources may include mtil- 
tiple computer display terminals located in different parts of 
a hospital, hard copies of medical images printed on film 
archived among thousands of images in a hospital film 
library or remote storage site, and volumes of journals 
located in the stacks of a hospital library. Also, the sources 
of data consulted by treating physicians may include medi- 
cal atlases containing thousands of MR and CT scans of a 
cadaver corresponding to photographic images of cross- 
sectional slices taken of various anatomical structures. 

Usually data from these atlases and other sources are not 
correlated with each other. A cadaver image in an atlas does 
not usually have the same geometry as a patient receiving 
treatment, so a physician must mentally fuse the available 
data which requires correlating the data retrieved from the 
various sources lo develop a treatment plan or lo provide 
information during medical procedures. The difficulties of 
fusing all of the available data increase if the physician must 
assimilate the various data types while rendering treatment. 

Hie World Wide Web ("WWW") has recently made vast 
amounts of data stored on local and remote computers easily 
accessible through a graphical computer interface. The 
WWW is a network of computers, connected by the Internet, 
sharing a common file stmcture and mark-up language for 
creating files. The two most prevalent languages used to 
create multimedia WWW files are the hypertext mark-up 
language ("HTML") and the virtual reality mark-up lan- 
guage ("VRML"). HTML is best suited for creating files 
with text and two-dimensional image data, whereas VRML 
is designed for creating files containing images of three- 
dimensional objects. Both languages provide an easy way to 
combine image, text, and sound data in files accessible by 
"point-and-click," computer mouse driven user interfaces 
called "browsers." 

A "browser" is a computer program that provides users 
access to files stored on the WWW. The browser displays 
files on a computer screen and can run programs, known as 
"applets," indicating links to data in other files on the WWW 
by, for example, underlining text or highlighting areas of an 
image. By selecting the underlined text or a highlighted 
image, the browser retrieves the linked data, allowing a user 
to view data stored on computers in the WWW without 
needing to know where the information is physically stored. 
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Files can be joined using these "hyperlinks," which give the 
name of the file along with an address for a computer storing 
the file. For example, the text or an image in a file stored on 
a computer in Switzerland can contain an embedded link to 

5 data stored on a computer in the United Slates. The WWW 
browser automatically recognizes the linked file data type, 
so the linked file can be an image, an audio clip, a video, or 
even an executable computer program. For example, if the 
linked data is an audio clip, the browser will load a program 

10 that takes the audio clip and plays it through the speakers of 
the user's computer. A browser usually mns on a computer 
referred to as a "client," while a computer known as a 
"server" hosts and produces WWW files requested by a 
client. 

15 In particular, the WWW serves as a useful tool for 
navigating through two- and three-dimensional image data. 
For example, an image can be displayed by the browser, and 
different parts of the image can be linked to different files. 
But, for the most part, this WWW capability is primarily 

20 used for providing simple menus of uncorrclated data avail- 
able on WWW computers. For example, a WWW computer 
will show an image of people, cars, and boats. By clicking 
on the image of people, a user can go to on-line chat sessions 
with people, or by clicking on a boat image, a user gets 

^ information about boats. 

The current technology is limited because there does not 
exist an information system that exploits the data navigation 
capabilities of the WWW to correlate data retrieved from 
diverse sources and then assimilate the data into a useful 
form. For example, the tools available for information 
gathering in the WWW environment include database search 
engines and expert systems that assist a user in describing 
the information sought. However, these tools only retrieve 
files corresponding to a particular term or pertaining to 
certain designated subject matter. The retrieved files are not 
correlated with one another. 

There is, therefore, a need for an information system that 
harnesses the power of the technology associated with the 
WWW and other similar image -based information retrieval 
systems lo produce assimilated composite data in a form that 
operators can readily use. 

SUMMARY OF THE INVENTION 

45 The present invention is directed to a method and appa- 
ratus for producing and accessing composite data containing 
co-registered and subject data. The method for automatically 
producing composite data includes several steps, performed 
by a server computer. The steps include: creating a mapping 

5Q relationship between the co -registered data and the subject 
data by mapping or deforming a template to fit the subject 
data; filtering the co-registered data; and producing com- 
posite data by mapping the filtered co -registered data to the 
subject data according to the mapping relationship. 

55 A method consistent with this invention is also directed to 
steps, performed in a client computer, including: requesting 
composite data from a server computer; transmitting the 
subject data to the server computer; receiving the requested 
composite data from the server computer; presenting the 

60 received composite data to an operator; and monitoring the 
operator's use of composite data. 

An apparatus consistent with this invention for automati- 
cally producing composite data containing co- registered 
data and subject data includes: stmcture for creating a 

65 mapping relationship between the co-registered data and the 
subject data by mapping or deforming a template to fit the 
subject data; structure for filtering the co-registered data; 
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and structure for producing composite data by mapping the 
filtered co-registered data to the subject data according to the 
mapping relationship. 

Another apparatus consistent with the present invention 
automatically presents an operator with composite data 
containing co-registered data and subject data. Such an 
apparatus includes: structure for requesting composite data 
from a server computer; structure for transmitting the sub- 
ject data to the server computer; structure for receiving the 
requested composite data from the server computer; struc- 
ture for presenting the received composite data to an opera- 
tor; and structure for monitoring the operator's use of the 
received composite data. 

Both the foregoing general description and the following 
detailed description are exemplary and explanatory and are 
intended to provide further explanation of the invention as 
claimed. 

DESCRIPTION OF THE FIGURES 

The accompanying drawings provide a further under- 
standing of the invention. They illustrate embodiments of 
the invention and, together with the description, explain the 
principles of the invention. 

FIG. 1 is a block diagram of an apparatus for automati- 
cally producing composite data containing co-registered 
data and subject data consistent with the present invention; 

FIG. 2 is a block diagram of another apparatus for 
automatically producing composite medical data containing 
co-registered medical data and patient data consistent with 
the present invention; 

FIG. 3 is flow diagram of a method for automatically 
producing composite data containing co-registered data and 
subject data consistent with the present invention; 

FIG. 4 is a schematic diagram of user interaction with the 
embodiment of the invention shown in the block diagram of 
FIG. 2; 

FIG. 5 is an illustration of co-registered medical data used 
in an embodiment of the present invention consistent with 
the block diagram of FIG. 2; 

FIG. 6 is a display produced in accordance with the 
embodiment of the present invention shown in the block 
diagram of FIG. 2; and 

FIG. 7 is a block diagram of a facility for providing 
composite data in accordance with an embodiment of the 
present invention. 

DETAILED DESCRIPTION OF THE 
INVENTION 

Reference will now be made in detail to the preferred 
embodiments of this invention, examples of which are 
illustrated in the accompanying drawings. 

To illustrate the principles of this invention, FIG. 1 shows 
subject data 100, template layer 102, deformation engine 
104, mapping engine 106, map 107, filtered, co-registered 
data 108, search and filter engine 110, links 112, and 
co-registered databases 114. The term "layer" denotes a 
grouping of data types represented in, for example, a data- 
base on a server computer. To produce composite data 109, 
deformation engine 104 deforms a template layer 102 to fit 
subject data 100 generating map 107. When subject data 100 
is multi-modal, multiple template layers can be utilized to 
correlate co-registered data 114 to subject data 100 as well 
as to correlate multi-modal subject data 100 to itself. A 
template layer contains reference data locations, or 
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landmarks, used to correlate template data with elements of 
subject data. Examples of landmarks include data represent- 
ing points, lines, surfaces, volumes, or other defining fea- 
tures in image data. 

5 Generally, deformation is the process of mapping one 
image to another image where both images represent the 
same stmcture but have different geometric proportions and 
orientations. During deformation, mathematical transforms 
are applied to the images that perform the equivalent of 
bending, stretching, and rotating template data to match 
subject data. For example, after deformation, template data 
in the form of a volume image of a generalized model of the 
human brain is manipulated so that the anatomical structure 
in this model has the same size, shape, and orientation as the 
subject data, the actual anatomy of a patient receiving 
treatment. There are many techniques available for auto- 
matically deforming one set of data to fit a target data set, 
including rule-based morphology, correlation of selected 
landmarks in each data set, and a technique fusing selected 
landmarks and image data. One example of such technique 

20 appears in U.S. patent application Ser. No. 08/678,628 filed 
Jul. 10, 1996, which is herein incorporated by reference. 

Once template layer 102 is deformed to fit subject data 
100, a mapping relationship, map 107, is established 
whereby mapping engine 106 maps co-registered data 114 to 

25 subject data 100 producing composite data 109. 
Co-registered data 114 represents a knowledge base provid- 
ing supplemental information about structures contained in 
subject data 100. The co-registered databases and template 
layer share a common coordinate system, so a data element 

3Q representing a position in one co-registered database is 
correlated with a data element representing that same posi- 
tion in each of the other co -registered databases. In an 
embodiment consistent with the present invention, 
co-registered data 114 is co-registered using, for example, 

35 the deformation techniques described above. ITie mapping 
relationship obtained from deforming template layer 102 to 
fit subject data 100 correlates the co -registered database 
coordinate system with a subject data coordinate system. 
Implementing this mapping relationship, mapping engine 

4Q 106 relates points in the subject data coordinate system to 
corresponding points in the co -registered database coordi- 
nate system, providing a dynamic connection between sub- 
ject data 100 and co-registered databases 114. 

Search and filter engine HO controls which elements of 

45 co-registered data 114 are mapped to subject data 100 and 
presented to an operator. Search and filter engine 110 can 
allow mapping engine 106 to map all or a subset of 
co-registered data 114 to subject data 100. Links 112 specify 
relationships among data elements across co -registered data- 

50 bases 114 which are used by search and filter engine 110 to 
assimilate co-registered data according to a service request 
by an operator including, for example, an indication of a 
region or regions of interest in subject data 100. The links 
t\2 may be formed using an appropriate database indexing 

55 strategy to assign key-word or concept search tags to asso- 
ciated data elements. 

FIG. 2 shows a preferred embodiment of the present 
invention for producing composite patient data for medical 
treatment. Collected patient data 200, includes, for example, 

60 MR image 236, CT image 238, and landmarks 240. Tem- 
plate layer 202 includes corresponding MR image 230, CT 
image 232, and landmark 234 templates. This embodiment 
also includes a deformation engine 204, mapping engine 
206, search and filter engine 210, links 212, and 

65 co-registered data 214. 

Co -registered data 214 mcludes a medical atlas 228, text 
226, computer programs 225 (such as applets), labels and 
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landmarks 224, images 222, audio and video clips 220, links transmits composite data 244 and map 248 to the client 

to computers located on the WWW 218, and treatment plans computer for presentation to an operator (step 318). The 

216. Co-registered data can also include co-registered sub- operator navigates the composite data by specifying a region 

ject data. The forgoing list of co-registered data types is only of interest in the patient data using a browser interface (step 

provided as an example of the types of data that arc useful s 320). The operator may also use the browser interface to 

in practicing this invention in the context of providing s^j^ct highlighted text or other specified segments of the 

medical treatment. Persons of ordinary skill wiU recognize composite dau, activating a link to a particular region of 

that many other data types may also be useful. One of ^^^^^ ^ ^^^^^ 

ordinary skill in the art will also recognize that two or more ^ x ^ 1 « 

of the co-registered daU types may reside in a single ,„ "^^P ^ """^ " "P"*'"' '° '"^"^ addiUonal 

database composite data (step 322). The server receives a request for 

- . a p *uj* J- additional co-registered data and preferably an associated 

FIG. 3 IS a flow diagram of a method for produang • .u «• * j ♦ a * *u 1 ^ 

, J , ^ , -.u ■ *■ A position m the patient data coordinates from the chent and 

composite medical data consistent with the mvention. An ^ . *i f- , • . j 

c , ■ , . .u- • • u- subsequently retrieves and transmits additional CO- registered 

operator of a system consistent with this invention wishing , , ^. . . . 

to have composite data for medical treatment first identified ''""^ ""?f„'"*Pn^'T-,' . f ' ""'iV /if 

. -ui 11 4 .• . J . tnn 1- . " engme 210, and hnks 212 to additional co-registered data 

and possibly collects patient data 200 using a client com- , .... - . ® 

, / . lAAs TT, 11 . J .• , J f . u 214. (repetition of steps 312-320). 

puter (step 300). The collected patient data must have a ^ ' 

modality (e.g., CT, MR, or X-ray) and protocol (e.g., slice ^n embodiment consistent with the present invention for 

thickness and resolution) that is compatible with a template '° ^eld links a radiologist's report to 

in template layer 202 radiological imagery of a patient. Preferably, in an area of 

TTie client computer then generates request 246 for com- ^° Ae radiologist's text report stating, for example, "in the left 

posite data 244 (step 304). Request 246 inchidcs, for par'etal-occipit<d region is a 1.2 cm hypodense lesion with 

example, an operator identifier, security screening an irregular border that does not enhance on contrast but is 

information, and treatment information. The client computer hypenntense on T2 . . . selecting the highlighted word 

also transmits or enables transmission of (from a radiologi- ^ activates a link to the paUent imagery which 

cal database, for example) collected patient data 200 and highlights the particular sub-region discussed m the report 

associated filtering context 242 to the server computer (step 'f.''^ Likewise, if the operator selects the lesion site in the 



306) 



displayed patient imagery, the link will be activated to 



n J- * 1- * * ,-\Acc •* display the section or sections of the text report that discuss 

Responding to client computer request 246 for composite *u i * j • * * * . 

j.-ii^.**!. * 1* c ■ the selected region of mterest. 

data 244, the server computer selects a template conformmg 30 

with this request (step 308). Example template, MR 230, Although the foregoing description of embodiments of the 

employed by the present invention when used for medical Pr«^°^ mvenUon specifically allocate certam operations to 

treatment includes a three-dimensional MR scan of the same ^^^^^^ ^'^^ ^^^^ computers, one of ordinary skill in the art 

slice thickness, resolution, and collection protocol as the recognize that the distribution of specific tasks between 

patient MR dataset 236. Associated with the selected tem- 35 ^^^^ computers can vary based on apphcalion 

plate imagery is a set of landmarks 234 identifying anatomi- requirements. Moreover, cmbodmients of the present mven- 

cal su^ctures in a region of interest for a particular surgical ^^^^ ^^^^ ^^^^''^ ^^^^^^ server computers are within the 

procedure. Next, deformation engine 204 fits the selected ^"'P*^ invention. Furthermore, it is also consistent 

template to patient data received from the chent computer. ^^^^ 'he present invention that the client and server tasks can 

The process of deforming the selected template to fit the 40 Performed m a single computer. 

patient data 200 creates a mapping relationship (map 248) ^° ^ preferred embodiment of the present invention, a 

relating template data space to a patient data space coordi- graphical user interface designed for browsing data presents 

nate system (step 310). Mapping engine 206 also uses map composite data 244 to the operator. The interface is prefcr- 

248 to relate a point in the patient data space coordinate ably executed on networked computers. Computer program 

system to an element of co -registered data 214. Once the 45 code that can be adapted to perform this browsing function 

mapping relationship is determined by deforming the includes Internet browsers designed to navigate the WWW, 

selected template, all co-registered data 214 can be mapped such as Netscape's Navigator and Microsoft's Explorer, and 

to patient data 200, Note that if muld-modal patient data is equivalent programs that support links to data stored on 

used with multiple corresponding templates, multiple maps networked computers. 

248 can be constructed that can then be used to correlate the 50 FIG. 4 illustrates operator interaction associated with 

multi-modal patient data with each other in addition to producing composite data in accordance with an embodi- 

correlating co-registered data 214. ment of the present invention. FIG. 4 shows a database 

Search and filter engine 210 controls how much of containing co-registered data 400 of several data types, 

CO- registered data 214 is included in the composite data. One including video 402, text 404, waveforms 406, programs 

reason for such control is that certain data types and/or data 55 407, still images 408, and segmentation labels 409; a map- 

elements in co-registered data 214 may not be relevant for ping engine 410; a map 412; a set of links 414 among 

some medical treatment. Search and filter engine 210 associated data elements across and within the co-registcrcd 

responds to filtering context 242 and selects co-registered databases; a patient data space coordinate system 416; and 

data elements as appropriate for this filtering context (step a universal ("atlas") coordinate system 418 common to all 

312) using links 212 to identify related data elements. The 60 data stored in co-registered database 400. A physician using 

filtering context can be derived from data provided by the this embodiment of the present invention selects any point in 

chent computer during the initial request for composite data patient data space coordinate system 416 to retrieve 

(step 304). A filtering context can also be derived from co-registered data 400 corresponding to the selected point, 

previously stored profiles or histories. The server then In an example illustrating co-registered data in one 

produces composite data using mapping engine 206 and map 65 embodiment of the invention, still image database 408 

248 to combine patient data and filtered, co-registered data contains MR images of a human head, database 406 contains 

(step 314), producing composite data 244. The server then recordings of waveforms produced by certain electrical 
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signals in the brain, video database 402 contains recorded 
motion picture images of neurosurgical procedures or tuto- 
rials for these procedures, text database 404 contains short 
descriptive paragraphs or full journal articles describing 
regions of the brain and related surgical plans, database 407 
contains programs for processing image data, and database 
409 contains segmentation maps outlining brain structures. 

The patient data space coordinate system 416 is a frame 
of reference for patient specific data. This coordinate system 
is provided by, for example, an MRI of a patient's head or 
the surgical field surrounding the patient during operation. 
Deformation engine 204 computes a mapping relationship 
relating template layer data points in atlas coordinate system 
418 to patient data points in patient data space coordinate 
system 416. Mapping engine 410 uses this computed map- 
ping relationship to transform co-registered data 400 
mapped to atlas coordinate system 418 to patient data space 
coordinate system 416. 

After mapping, a physician has available during a surgical 
procedure composite data adapted to the patient's anatomy. 
This composite data is a representation of (1) a patient's 
anatomy comprising patient specific data acquired before or 
during a medical procedure, and (2) data from one or more 
of the co-registered databases 400. 

Map 412 provides a virtual grid overlaying the patient 
data space coordinate system 416 allowing an operator to 
position a pointing device in the patient data to retrieve 
co-registered data. Selecting a position in map 412 retrieves 
co-registered data correlated with the selected position by 
mapping engine 410 through links 414. 

In one embodiment of the invention, map 412 contains a 
number of positions corresponding to the number of posi- 
tions in patient data space coordinate system 416 detectable 
by a surgical navigation system (see, e.g., U.S. Pat. No. 
5,383,454). A map position is selected according to the 
location of a surgical probe in patient data space coordinate 
system 416 during a medical procedure. For example, during 
neurosurgery a surgeon placing the probe at a patient *s 
ventricle activates a map position corresponding to the probe 
position in the ventricle. The activated map position is 
communicated to mapping engine 410 which queries 
co-registered databases 400 for data corresponding to the 
map position in the ventricle. This corresponding 
co-registered data is deformed to fit the patient's anatomy 
and combined with patient specific data giving the surgeon 
composite data related to the patient's ventricle, containing 
more information than the patient data alone. 

A further illustration of the types of data a physician may 
require during neurosurgery or during surgical plaiming is 
shown in FIG. 5. This figure contains illustrations of five 
data types available to a surgeon in an embodiment of the 
present invention including a cross-sectional image 502, a 
medical journal article 504, electroencephalograph wave- 
forms 506, computer programs 507, video images of the 
brain 508, and a segmentation map identifying the different 
regions of the brain 510. Because these data sources have 
been co-registered to a common atlas coordinate system, a 
point, such as point 500 in tbe cerebellum in brain image 502 
from database 408, has a corresponding point in each of the 
other data types in co-registered database 400. For example, 
point 500 in text database 404 corresponds to article 504 on 
new surgical techniques involving the cerebellum. Point 500 
in waveform database 406 corresponds to recorded wave- 
forms 506 produced by the brain at this location. Point 500 
in program database 407 corresponds to applet program 507, 
which is used to provide enhanced visualization of brain 
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image 502. Point 500 in video database 402 corresponds to 
video clips 508 of the brain at this location. Point 500 in 
segmentation map database 409 corresponds to a point 
within segmentation map 510. 

Each of these examples of data need not be acquired from 
the patient currently receiving treatment. For example, the 
data may come from digital anatomical atlases, libraries of 
cadaver images, or research databases produced by projects 
such as the "Visible Human" research sponsored by the 
10 National Library of Medicine. Data available in a 
co-registered database would include, for example: 

1 . Anatomic 

Magnetic Resonance Imaging 
Computed Tomography 
Magnetic Resonance Angiography 
Ultra-Sound 

Slice photographic images 
Sulci/Gyri traces 

2. Functional 

Positron Emission Tomography 
25 Single Photon Emission Computed Tomography 
Functional Magnetic Resonance images 
Electroencephalograph 
Magnet oencephalography 

3. Symbolic 

Structure name 
Structure size 
Structure function 
35 Structure related text cues 
Structure related video cues 
Structure related audio cues 
Structure related labels 
Histology 

Morphological data 

4. Multimedia 

Video Footage of procedures 
45 Training Videos 

Conference, Journal Articles 
Statistics 

5. Computer Programs 

Applets 
Data Analysis 

Automated Structural Segmentation 
Image Enhancement and Visualization 

55 These data need not be located in a single database. One 
of ordinary skill in the art will recognize that individual 
co-registered databases may be distributed among several 
databases accessible through, for example, local area com- 
puter networks or wide area computer networks connecting 

60 local and distributed computers. The combination of the 
different data components of the co -registered databases 
produces a generic data model of human anatomy where 
each data element is correlated through a conunon coordi- 
nate system with corresponding data elements of other 

65 types. When planning or providing medical treatment, the 
invention produces composite data linking co-registered 
data of a generic model to the specific anatomy of a patient 
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receiving treatmenl. Examples of other embodiments con- 
sistent with the present invention for producing composite 
data for medical diagnosis, planning, and treatment include, 
but are not limited to, the following. 

1. Diagnostic Radiology — co-registered patient magnetic 
resonance, X-ray, and/or computed tomography imag- 
ery arc linked to text data such as radiologists'reporls, 
patient case history files, and relevant conference/ 
journal articles. Sequential scans of a patient arc 
co-registered for tracking the growth or reduction of 
lesions. Data analysis programs arc linked to the com- 
posite data for computation of quantitative data mea- 
surements for planning and monitoring of treatment 
progress. Co-registered multi-modal patient image data 
and relevant co-registered data are presented in a 
common, easy-to-use presentation scheme. 

2. Radiation Treatment Planning — Three-dimensional 
segmented atlases are mapped to patient data to pro- 
duce an object-based model of lesions, targets, and 
major organs and other critical structures. The patient 
data with associated object information is utilized by a 
treatment planning program for computing optimized 
radiation delivery strategies from target and critical 
strucmre information. 

3. Neurosurgical Targeting — cranial patient imagery is 
mapped to neurosurgical atlas information containing 
coordinates and shapes of surgical targets and sur- 
rounding neuroanatomic structures. Structural informa- 
tion is linked to audio files for use in -surgery with 
microrecording probes. Links to statistical databases 
provide information relating regions of interest to pro- 
cedures and success rates. 

FIG. 6 shows how one embodiment of the present inven- 
tion presents composite data to an operator on a computer 
display having several windows. Window 604 contains 
patient data in the form of a three-dimensional MR brain 
scan. Windows 602, 606, and 608 contain axial, sagittal, and 
coronal photographic section data, respectively, from the 
Visible Human data set, which are co-registered to the 
patient data by deformation engine 204. Window 610 pre- 
sents CO- registered atlas data to an operator. By positioning 
cross-hairs 612 in Window 604 at a desired point 600, the 
corresponding point 600 in each of the images in windows 
602, 606, and 608 is identified automatically by the location 
of cross-hairs in those windows. Here, for example, the 
operator selected point 600 in window 604 ctorresponding lo 
a region of the brain known as the putamen. Atlas data 
containing views of the putamen are displayed in window 
610 with highlighted pointers indicating the putamen in each 
of the atlas images. The operator can also choose to play a 
movie showing video images of the putamen by pushing 
"play movie" button 614. Alternatively the operator may 
select the word "putamen" in window 610 and cross- hairs 
612 will indicate the position of the putamen in patient data 
window 604 and the Visible Human "atlas" data windows 
602, 606, 608. 

FTG. 7 is a block diagram of a facility consistent with the 
present invention for providing composite data across a 
computer network to customers imder a service contract. In 
FIG. 7, solid lines indicate a path for both control and data 
flow and dotted lines indicate data flow only. Facility 700 is 
preferably connected to a wide area network, such as Inter- 
net 701, through firewall 702. Firewall 702 is a computer 
that monitors all data traf&c into and out of facility 700 to 
prevent unauthorized access to the facility. World Wide Web 
page 704 provides a graphical user interface to access 
facility 700. Facility 700 also includes customer account 
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manager 710, which controls functions available to custom- 
ers with service contracts authorizing access to facility 700. 

User login authentication is performed by customer 
account manager 710 and control is passed to one of three 

5 processes, service request manager 706, customer database 
manager 708, or results manager 712 depending on the 
service that the customer diooses. Customers that wish to 
initiate a new request for composite data are passed to 
service request manager 706. After successful completion of 

10 a composite data request, the customer's accoimt is billed 
and the status of any pending requests is provided. Custom- 
ers that wish to view the composite data generated in 
response to the request are passed to results manager 712. 
Information pertaining to a customer*s account (e.g., billing 

15 information, changing passwords, user preferences, etc.) 
may be obtained by submitting queries to customer database 
manager 708. 

Service request manager 706 initiates service requests and 
controls the computational operations required for anatomic 

20 mapping. The progress of a service request is tracked and 
reported to customer account manager 710. Customer data- 
base manager 708 administers a database that contains 
customer account data (not shown). Customer database 
manager 708 is responsible for controlling and backing up 

25 the customer database and it also processes queries from 
customer account manager 710, service request manager 
706, and results manager 712. Results manager 712 inte- 
grates the results generated by service request manager 706 
with context-spedfic medical knowledge. Results manager 

30 712 receives information from search and filter engine 720 
and mapping engine 722 specific to application require- 
ments. Some results may be provided as visual representa- 
tions (e.g., mapped segmented structures in a brain image) 
while others may be expressed in numeric form (e.g.. 

35 coordinates of a neurosurgical target). 

Preprocessor 714 checks patient data associated with a 
service request to make sure that the patient data is in the 
correct format and that the service request is appropriate. 
Support personnel 724 confinn the check performed by 

40 preprocessor 714. Any inconsistencies or anomalies that are 
found are reported lo service request manager 706. 
Similarly, post processor 718 checks the results of a service 
request. Support personnel 724 confirm the check performed 
by preprocessor 718, Any inconsistencies or anomalies that 

45 are fotmd are reported to service request manager 706. 

Facility 700 also includes deformation engine 716, search 
and filter engine 720, and mapping engine 722. Deformation 
engine 716 computes atlas-to-patient transformations 
requested by service request manager 706. Search and filter 

50 engine 720 processes customer, patient, and procedure con- 
textual data and integrates relevant atlas information accord- 
ing to application requirements. Both results manager 712 
and service request manager 706 initiate search and filter 
engine 720 operations. Following a request from results 

55 manager 712, mapping engine 722 applies the results of the 
deformation process mapping an atlas to the patient data. 

Patient database manager 726 administers a database that 
contains patient data and corresponding transformations 
computed by deformation engine 716. Patient database 

60 manager 726 serves queries from deformation engine 716, 
search and filter engine 720, and mapping engine 722 and is 
also responsible for controlling and backing up the patient 
database (not shown). 
Atlas database manager 728 administers a database that 

65 contains the atlas data (not shown). Atlas database manager 
728 serves queries from deformation engine 716, search and 
filter engine 720, and mapping engine 722 and is also 
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responsible for controlling and backing up the atlas data- 
base. Atlas database manager 728 can also perform and/or 
manage the indexing of the co -registered data databases. 

While there has been illustrated and described what arc at 
present considered to be preferred embodiments and meth- 
ods of the present invention, persons skilled in the art will 
understand that various changes and modifications may be 
made, and equivalents may be substituted without departing 
from the scope of the invention. 

In addition, many modifications may be made to adapt a 
particular element, technique or implementation to the 
teachings of the present invention without departing &om 
the central scope of the invention. For example, disclosed 
elements may be implemented in hardware, computer pro- 
gram code, or a combination of both hardware and computer 
program code. Moreover, elements depicted and described 
separately may be combined and implemented in a single 
element. Therefore, this invention is not limited to the 
particular embodiments and methods disclosed, but includes 
all embodiments falling within the scope of the appended 
claims. 

We claim: 

1. A method for automatically producing composite data 
containing co -registered data and subject data, comprising 
the steps, performed in a server computer having a mapping 
engine, a search and filter engine, and a deformation engine, 
of: 

creating a mapping relationship between the co-registered 

data and the subject data by mapping or deforming a 

template to fit the subject data; 
filtering the co-registered data; and 
mapping the filtered co-registered data to the subject data 

according to the mapping relationship to produce the 

composite data. 

2. The method of claim 1, wherein said co-registered data 
contains co -registered subject data. 

3. The method of claim 1, wherein the step of mapping 
includes the substep of: 

relating a point in a subject data space coordinate system 
to an element of co-registered data. 

4. The method of claim 1, wherein the step of mapping 
includes the substep of: 

relating an element of co-regtstered data to a point in a 
subject data space coordinate system. 

5. The method of claim 1, wherein the method further 
includes the step of: 

receiving a request for said composite data from a chent 
computer. 

6. The method of claim 1, wherein the method further 
includes the step of: 

receiving a request, including a filtering context, for said 
composite data from a client computer. 

7. The method of claim 1, further comprising the step of: 
communicating said composite data to a client computer. 

8. The method of claim 1, further comprising the step of: 
communicating a map representing the relationship 

between said subject data and said co-registered data to 
a client computer. 

9. The method of claim 1, farther comprising the step of: 
filtering said co-registcred data according to a filtering 

context. 

10. A method for automatically presenting an operator 
with composite data containing co-registered data and sub- 
ject data, comprising the steps, performed in a client 
computer, of: 
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requesting composite data from a server computer; 
transmitting subject data corresponding to a first subject 

to said server computer; 
co-registering said subject data corresponding to the first 
^ subject with subject data corresponding to a second 

subject to produce said requested composite data; 
receiving said requested composite data from said server 

computer; 

JO presenting said received composite data to an operator; 
and 

monitoring the operator's use of said received composite 
data. 

U. The method of claim 10, wherein the presenting step 
15 includes the substep of: 

displaying said received composite data as an image. 

12. The method of claim 10, wherein the presenting step 
includes the substep of: 

displaying said received composite data as text 

13. The method of claim 10, wherein the presenting step 
includes the .substep of: 

generating sound from said received composite data. 

14. The method of claim 10, wherein the presenting step 
^ includes the substep of: 

displaying said received composite data as a video clip. 

15. The method of claim 10, wherein the presenting step 
includes the substep of: 

executing a program included in said received composite 
30 data. 

16. The method of claim 10, wherein the presenting step 
includes the substep of: 

displaying said received composite data in a browser 
having a graphical operator interface. 
^5 17. An apparatus having a mapping engine, a deformation 
engine, and a search and filter engine for automatically 
producing composite data containing co-registered data and 
subject data, comprising: 
means for creating a mapping relationship between the 
co-registered data and the subject data by mapping or 
deforming a template to fit the subject data; 
means for filtering the co-registered data; 
means for mapping the filtered co -registered data to the 
45 subject data according to the mapping relatioaship to 
produce the composite data. 

18. The apparatus of claim 17, wherein said co-registered 
data contains co-registered subject data. 

19. The apparatus of claim 17, wherein the means for 
50 creating a mapping relationship comprises; 

means for relating a point in a subject data space coor- 
dinate system to an element of co-registered data. 

20. The apparams of claim 17, wherein the means for 
mapping further comprises: 

55 means for relating an element of co-registered data to a 
point in a subject data space coordinate system. 

21. The apparatus of claim 17, further comprising: 
means for receiving a request for said composite data 

from a client computer. 

22. The apparatus of claim 17, further comprising: 
means for processing a filtering context; and 

means for receiving a request, including a filtering context 
for said composite data from a cUent computer. 
^5 23. The apparatus of claim 17, further comprising: 

means for communicating said composite data to a client 
computer. 
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24. The apparatus of claim 17, further cotaprising: 
means for communicating a map representing the rela- 
tionship between said subject data and said 
co-registered data to a client computer. 

25. The apparatus of claim 17, wherein the means for 
producing composite data comprises: 

means for filtering said co -registered data according to a 
filtering context. 

26. An apparatus for automatically presenting an operator 
with composite data containing co-registered data and sub- 
ject data, comprising: 

means for requesting composite data from a server com- 
puter; 

means for transmitting subject data corresponding to a 

first subject to said server computer; 
means for co-registering said subject data corresponding 

to the first subject with subject data corresponding to a 

second subject to produce said requested composite 

data; 

means for receiving said requested composite data from 

said server computer; 
means for presenting said received composite data to an 

operator; and 

means for monitoring the operator's use of said received 
composite data. 

27. The apparatus of claim 26, wherein the means for 
presenting comprises: 

means for displaying said received composite data as an 
image. 

28. The apparatus of claim 26, wherein the means for 
presenting comprises: 

means for displaying said received composite data as text. 

29. The apparattis of claim 26, wherein the means for 
presenting comprises: 

means for generating sound fi:om said received composite 
data. 

30. The apparatus of claim 26, wherein the means for 
presenting comprises: 

means for displaying said received composite data as a 
video clip. 

31. The apparatus of claim 26, wherein the means for 
requesting comprises: 

means for browsing data stored on a computer presented 
to the operator in a graphical form. 

32. The apparatus of claim 26, wherein the means for 
presenting comprises: 

means for browsing data stored on a computer presented 
to the operator in a graphical form. 

33. The apparatus of claim 26, wherein the means for 
presenting comprises: 

means for executing a program included in said received 
composite data. 

34. A method for automatically producing composite 
medical data containing co-registered medical data and 
patient data, comprising the steps, performed in a server 
computer having a mapping engine, a search and filter 
engine, and a deformation engine, of: 

creating a mapping relationship between the co-registered 
medical data and the patient data by mapping or 
deforming a template to fit the patient data; 
filtering the co -registered medical data; and 
mapping the filtered co -registered medical data to the 
patient data according to the mapping relationship to 
produce the composite medical data. 
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35. The method of claim 34, wherein said co-registered 
medical data contains co -registered patient data. 

36. The method of claim 34, wherein the step of mapping 
includes the subslcp of: 

s relating a point in a patient data space coordinate system 
to an element of co-registered data. 

37. The method of claim 34, wherein the step of mapping 
includes the substep of: 

relating an element of co-registered medical data to a. 
point in a patient data space coordinate system. 

38. The method of claim 34, wherein the method further 
includes the step of: 

receiving a request for said composite medical data from 
a client computer. 

39. The method of claim 34, wherein the method further 
includes the step of: 

receiving a request, including a filtering context, for said 
composite medical data from a cUent computer. 
20 40. The method of claim 34, further comprising the step 
of: 

communicating said composite medical data to a client 
computer. 

41. The method of claim 34, further comprising the step 

25 of: 

communicating a map representing the relationship 
between said patient data and said co-registered medi- 
cal data to a client computer 

42. The method of claim 34, further comprising the step 

30 of: 

filtering said co-registered medical data according to a 
filtering context. 

43. A method for automatically presenting an operator 
with composite medical data containing co -registered medi- 
cal data and patient data, comprising the steps, performed in 
a chcnt computer, of: 

requesting composite medical data from a server com- 
puter; 

40 transmitting patient data corresponding to a first patient to 
said server computer; 
co-registering said patient data coaesponding to the first 
patient with patient data corresponding to a second 
patient to produce said requested composite medical 

45 data; 

receiving said requested composite medical data from 

said server computer; 
presenting said received composite medical data to an 

operator; and 

monitoring the operator's use of said received composite 
medical data. 

44. The method of claim 43, wherein the presenting step 
inchides the substep of: 

displaying said received composite medical data as an 
image. 

45. The method of claim 43, wherein the presenting step 
includes the substep of: 

displaying said received composite medical data as text. 

46. The method of claim 43, wherein the presenting step 
includes the substep of: 

generating sound from said received composite medical 
data, 

47. The method of claim 43, wherein the presenting step 
65 includes the substep of: 

displaying said received composite medical data as a 
video clip. 
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48. The method of claim 43, wherein the presenting step 
includes the substep of: 

executing a program included in said received composite 
medical data. 

49. The method of claim 43, wherein the presenting step 
includes the substep of: 

displaying said received composite medical data in a 
browser having a graphical operator interface. 

50. An apparatus having a mapping engine, a deformation 
engine, and a search and filter engine for automatically 
producing composite medical data containing co-registered 
medical data and patient data, comprising: 

means for creating a mapping relationship between the 
CO- registered medical data and the patient data by 
mapping or deforming a template to fit the patient data; 

means for fiQtering the co-registered medical data; 

means for mapping the filtered co-registered medical data 
to the patient data according to the mapping relation- 
ship to produce the composite medical data. 

51. The apparatus of claim 50, wherein said co-registered 
medical data contains co-registered patient data. 

52. The apparatus of claim 50, wherein the means for 
creating a mapping relationship comprises: 

means for relating a point in a patient data space coordi- 
nate system to an element of co- registered medical 
data. 

53. The apparatus of claim 50, wherein the means for 
mapping further comprises: 

means for relating an element of co-registered medical 
data to a point in a patient data space coordinate 
system. 

54. The apparatus of claim 50, further comprising: ' 
means for receiving a request for said composite medical 

data from a cUent computer. 

55. The apparatus of claim 50, further comprising: 
means for processing a filtering context; and 

means for receiving a request, including a filtering context 
for said composite medical data from a client computer. 

56. The apparatus of claim 50, further comprising: 
means for communicating said composite medical data to 

a client computer. 

57. The apparatus of claim 50, further comprising: 
means for communicating a map representing the rela- 
tionship between said patient data and said 
co-registered medical data to a client computer. 

58. The apparatus of claim 50, wherein the means for 
producing composite medical data comprises: 

means for filtering said co-registered medical data accord- 
ing to a filtering context. 

59. An apparatus for automatically presenting an operator 
with composite medical data containing co-registered medi- 
cal data and patient data, comprising: 

means for requesting composite medical data from a 

server computer; 
means for transmitting patient data corresponding to a 

first patient to said server computer, 
means for co -registering said patient data corresponding 

to the first patient with patient data corresponding to a 

second patient to produce said requested composite 

medical data; 

means for receiving said requested composite medical 

data from said server computer; 
means for presenting said received composite medical 

data to an operator; and 
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means for monitoring the operator's use of said received 
composite medical data. 

60. The apparatus of claim 59, wherein the means for 
presenting comprises: 

means for displaying said received composite medical 
data as an image. 

61. The apparatus of claim 59, wherein the means for 
presenting comprises: 

means for displaying said received composite medical 
data as text. 

62. The apparatus of claim 59, wherein the means for 
presenting comprises: 

means for generating sound from said received composite 
medical data. 

63. The apparatus of claim 59, wherein the means for 
presenting comprises: 

means for displaying said received composite medical 
data as a video clip. 

64. The apparatus of claim 59, wherein the means for 
requesting comprises: 

means for browsing data stored on a computer presented 
to the operator in a graphical form. 

65. The apparatus of claim 59, wherein the means for 
presenting comprises: 

means for browsing data stored on a computer presented 
to the operator in a graphical form. 

66. The apparatus of claim 59, wherein the means for 
presenting comprises: 

means for executing a program included in said received 
composite medical data. 

67. An apparatus for automatically presenting an operator 
with composite medical data containing co-registered medi- 
cal data and patient data, comprising: 

a surgical navigation system for requesting composite 

medical data from a server computer: 
means for transmitting the patient data to said server 

computer; 

means for receiving said requested composite medical 

data from said server computer; 
means for presenting said received composite medical 

data to an operator; and 
means for monitoring the operator's use of said received 

composite medical data. 

68. An apparatus for automatically presenting an operator 
with composite medical data containing co-registered medi- 
cal data and patient data, comprising: 

means for requesting composite medical data from a 

server computer; 
means for transmitting the patient data to said server 

computer; 

means for receiving said requested composite medical 

data from said server computer; 
a surgical navigation system for presenting said received 

composite medical data to an operator; and 
means for monitoring the operator's use of said received 

composite medical data. 

69. A method for automatically presenting an operator 
with composite medical data containing co-registered medi- 
cal data and patient data, comprising the steps, performed in 
a computer, of: 

co-registering data corresponding to a first patient with 
data corresponding to a second patient producing com- 
posite medical data; 

linking the composite medical data to text contained in a 
medical report; 
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receiving operator input selecting a word in said medical 
report; and 

presenting to the operator said linked composite medical 
data corresponding to said selected word. 

70. A method for automatically presenting an operator 
with text from a medical report corresponding to composite 
medical data containing co-rcgistercd medical data and 
patient data, comprising the steps, performed in a computer, 
of: 

CO -registering data corresponding to a first patient with 
data corresponding to a second patient producing com- 
posite medical data; 

linking the composite medical data to text contained in the 
medical report; 

receiving operator input selecting a point in an image 
representing the composite medical data; and 

presenting to the operator said linked text corresponding 
to said selected point. 

71. A method for automating diagnosis, planning 
treatment, or monitoring medical treatment progress using 
composite medical data containing co-registered medical 
data and patient data, comprising the steps, performed in a 
computer, of: 

linking a data analysis computer program to the compos- 
ite medical data; 

computing a measure using said data analysis computer 
program; and 

presenting said measure to an operator of said computer 
for tise in diagnosis, planning treatment, or monitoring 
medical treatment progress. 
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72. A method for automating radiation treatment planning 
using composite medical data containing co -registered 
medical data and patient data, comprising the steps, per- 
formed in a computer, of: 

mapping a three-dimensional segmented atlas to the 
patient data; 

generating a model of an anatomical structure and a 
pathological structure in the patient data; and 
J generating a radiation treatment plan using said model. 

73. A method for automatically selecting a target using 
composite medical data containing co-registered medical 
data and patient data, the co -registered data including a 
target atlas, comprising the steps, performed in a computer, 

, of: 

generating a map by deforming a template to said patient 
data; 

creating target information by mapping said target atlas to 
said patient data using said map; and 
J presenting said target information to an operator of said 
computer for planning treatment. 

74. A facility for providing composite data including 
subject data and co-registered data, the facility comprising: 

a deformation engine for computing a data transformation 
^ responsive to a request for composite data; 

a search and filter engine for extracting data from an atlas 
database; and 

a mapping engine for mapping data extracted by said 
^ search and filter engine to said subject data. 

4' * * >¥ * 
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